Thanks @mxinden!

As another alternative, instead of setting KeepAlive::No once we identified the remote, we could as well set KeepAlive::Until(Instant::now() + Duration::from_secs(10)), thus giving the remote a chance to identify us.

Another idea:
If I understand it correctly, when two peers connect they both initiate their outbound substream request after the initial delay, which is usually the same for both peers. The current problem is just that the negotiation of one direction resolves already while the other one is still ongoing, and then the connection is closed, right?
What do you think about instead of having our keep_alive property we could have two flags is_inbound_ongoing and is_outbound_pending, and then implement

fn connection_keep_alive(&self) -> KeepAlive {
    if self.is_outbound_ongoing || self.is_inbound_ongoing {
        KeepAlive::Yes
    } else {
       KeepAlive::No
   }
}

That way in the above case when swarm1 identified the remote, it would still wait for their own inbound request to resolve before closing.
Note that this does not fix it if the initial delay is different, or if one side resolves before the other side even got the chance to init the outbound subtream request. Maybe not a problem in real-world systems where the negotiation process takes much longer, but for our tests it might make sense to still additionally add your above fix within the test.

Edit: I just noticed that my above idea does not make sense, so please ignore.
I don't think we need the 10s delay outside of our test case. In real-world system we won't ever run into this issue, right? Assuming that both peers usually initiate their outbound substream request at the same time, one side will always first send out their own outbound substream request before receiving the inbound one and responding to it. So it is quite unlikely that one peer will receive the response to their outbound request, before receiving the remote's request, right?

Edit: I just noticed that my above idea does not make sense, so please ignore.
I don't think we need the 10s delay outside of our test case. In real-world system we won't ever run into this issue, right? Assuming that both peers usually initiate their outbound substream request at the same time, one side will always first send out their own outbound substream request before receiving the inbound one and responding to it. So it is quite unlikely that one peer will receive the response to their outbound request, before receiving the remote's request, right?

I am assuming the same, and I have not seen an indication for this being a problem in the wild.

In case we ever do come across this, we can still introduce the KeepAlive::Until(10_SECONDS).